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METHOD AND APPARATUS FOR USER RECOGNITION USING CCD 

CAMERAS 

BACKGROUND OF THE INVENTION 

1 . Cross Reference to Related Applications 

[0001] This application is a continuation-in-part of U.S. patent application Serial No. 
09/512,489 which was filed with the U.S. Patent and Trademark Office on February 24, 
2000. 

2. Field of the Invention 

[0002] The present invention relates to recognition systems, and more 
particularly to a method and apparatus for user recognition to grant access to computer 
systems using charged coupled device (CCD) cameras. 

3. Description of the Related Art 

[0003] Computer systems often have password-based recognition systems for 
identifying the user and their authorization to access the system. The passwords are 
generally user selected and provide a "what you know" type strategy to access the 
system, i.e., if you know the password you can access the system. These password 
systems are relatively unreliable because the user may disclose it to anyone else, either 
intentionally or unintentionally, so they may access the particular computer system 
and/or service provided by the same. In addition, a user may write down his password 
so as to not forget it, thereby providing the opportunity for anyone to find and use it. A 



2 



By Express Mail # EV363605474US 

password is easily learned by actual disclosure or by being viewed by another while the 
user is typing it in to access the system. Thus, it is clear that a written password is a 
large security risk as it may provide several people with access to a system that only 
one person has been authorized to use. 

[0004] A more secure recognition system focuses on who the user is, i.e., 
characteristics of the user, as opposed to what the user knows. It would therefore be 
preferable to provide a recognition system that is not based on what the user knows, but 
rather is based on who the user is. 

[0005] One example of a method of determining who the user is, is fingerprint 
identification systems. These systems require the user to place their finger or hand on a 
dedicated scanner which will scan the finger or handprint of the individual and compare the 
scanned print with those stored in memory. Upon approval of the scanned print, the user 
will be provided access to the computer system. It is understood that that the level of 
access provided may be limited or tailored to the specific individual. 
[0006] Other known systems utilize a single camera that takes a two-dimensional 
image of the user's face and compares it with stored 2-D images in memory. For 
example, a system based on Connectix QuickCam is not capable of seeing a 3- 
dimensional object and therefore are limited to 2-dimensional applications. Thus, it is 
apparent that the primary drawback of such security system is that it must be capable of 
handling a 2-D photograph of the authorized user. 
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[0007] Thus, it would be desirable to provide a user recognition system that is 
capable of using 3-dimensional models of the user, and would therefor provide a more 
secure system. 
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SUMMARY OF THE INVENTION 

[0008] It is therefore an object of the invention to provide a method by which the 
user's identity is recognized by the system being accessed without requiring the user to 
input a password. 

[0009] It is yet another object of the invention to provide a method for recognizing 
a user's identity for enabling access to a computer system that is based on a who the 
user is rather than what the user knows (i.e., an input password). 
[0010] These and other objects are achieved in accordance with the method for 
recognizing a user of the present invention wherein at least two 2-dimensional still 
images of the user are obtained from at least two different angles of view. A 3- 
dimensional model of the user's face is then generated using the obtained images and the 
user's facial shape is determined using the generated 3-dimensional model. The 
determined facial shape is compared with user profile information stored in memory. The 
user profile information includes data relating to the facial shape of authorized users. The 
comparison with the stored user profile information results in a determination as to whether 
the determined facial shape matches the profile information stored in the memory. Access 
to the system is granted when it is determined that the obtained facial shape information 
matches that of a stored authorized user's profile. 

[0011] Other objects and features of the present invention will become apparent 
from the following detailed description considered in conjunction with the accompanying 
drawings. It is to be understood, however, that the drawings are designed solely for 
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purposes of illustration and not as a definition of the limits of the invention, for which 
reference should be made to the appended claims. 
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BRIEF DESCRIPTION OF THE DRAWINGS 

[0012] In the drawings wherein like reference numerals denoted similar elements 
throughout the several views: 

Figure 1 is a front view of a display monitor having recognition cameras 
according to an embodiment of the invention; 

Figure 1a is a front view of a mobile phone having recognition cameras 
according to another embodiment of the present invention; 

Figure 1b is a front view of a personal digital assistant having recognition 
cameras according to another embodiment of the present invention; 

Figure 2 is a front view of a display monitor having recognition cameras 
according to another embodiment of the invention; 

Figure 3a is a picture of a user taken by one of the cameras with a 
projected light according to an embodiment of the invention; 

Figure 3b is a picture of the user taken by the other of the cameras with 
the projected light according to an embodiment of the invention; 

Figures 4a, 4b and 4c are three-dimensional representations of the 
surface reconstruction of the user's face shown as a cloud of data points from various 
angles; 

Figure 5 is a normal picture of the user's face used to obtain facial texture 
of the user; and 

Figure 6 is a flow diagram illustrating the method of recognizing an 
authorized user according to the invention. 
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DETAILED DESCRIPTION OF THE PRESENTLY PREFERRED EMBODIMENTS 

[0013] Figure 1 shows a first embodiment of the user recognition system 
according to the invention. In this embodiment, the display monitor 12 includes a light 
source 18a and two charged couple device (CCD) cameras 14a and 16a. Cameras 14a 
and 16a may be any other suitable known camera, such as, for example, digital still or 
video cameras. The light source 18a projects a structured light onto the target, which in 
this case is the user's face as they are situated in front of display 12. The use of 
structured light projected onto the user's face facilitates the facial reconstruction 
process and makes such process more reliable by providing an accurate frame of 
reference in the imaging system. A computer or central processing unit 20 can be local 
to display monitor 12 or can be part of a network of connected computers. Cameras 
14a and 16a and light source 18a are connected to the computer 20 (either directly, or 
indirectly through a network of connected computers --e.g., a local area network, or a 
global network such as the internet). 

[0014] In another embodiment, the cameras and light projection apparatus may 

s 

be incorporated into a hand held device. For example, Fig. 1a shows a mobile 
telephone 120 a mobile telephone having a light source 118 and two cameras 114 and 
116. Fig. 1b shows a personal digital assistant (PDA) 220 also having a light source 
218 and two cameras 214 and 216. One of ordinary skill will recognize that current 3- 
dimensional scanning circuitry is small enough for incorporation into such devices and 
may be used as the imaging cameras 114, 116, 214, and 216 for such hand held 
apparatus. The hand held device 120, 220 is capable of communication with a global 
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network of computers (e.g., internet) 125, 225, either through wireless or wired protocol, 
such that processing of the received image and 3-dimensional reconstruction can be 
performed at a server 130, 230 connected to the network 125, 225 and re-transmitted to 
the hand held device 120, 220. Because of size and/or economics, the handheld device 
120, 220 may have only one camera. In that case, the camera is required to take two 
pictures from different angles. To automatically determine the angle between the two 
images, the handheld device may include a compass or other device for determining the 
orientation of the handheld device 120, 220. Alternatively, the user may be required to 
take two pictures from predetermined view points, i.e., a direct passport pose and a 
profile picture. Furthermore, Figs. 1a, 1b show the cameras on the same side of the 
device as the display and input device. However, the cameras may also be arranged 
on another side of the handheld device such as, for example, facing away from the 
display. Furthermore, the light source 1 18, 218 is also optional. 
[0015] Figure 2 shows another embodiment in which cameras 14b and 16b and 
light source 18b are integrated into the display monitor itself. Computer 20 includes a 
memory that can be any suitable memory storage device capable of retaining user 
profile information that includes 3-dimensional images of the user's face and facial 
texture information relating to the authorized users of the system. Examples of suitable 
memory devices would include random access memory (RAM), read only memory 
(ROM), erasible programmable read only memory devices (EPROMS), and any other 
magnetic or optical media such as hard and floppy disk drives or CD-ROMS (read only, 
recordable, writable or otherwise). 
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[0016] Figures 1, 1a, 1b, and 2 show the respective cameras on opposing sides 
of display 12. In accordance with an embodiment of the invention, the cameras should 
be aligned in the same horizontal plane with respect to each other. The vertical position 
of the cameras with respect to the display is not of integral importance and may be 
changed provided both cameras remain on the same horizontal plane, and the angles at 
which the still images are obtained by the cameras are slightly different with respect to 
the user being imaged. As the two cameras are placed in different places along the 
same horizontal plane, the points closer to the camera (i.e., on the user's face) appear 
to move left and right in the images, more than the points that are placed closer to the 
background of the user. Other contemplated embodiments may use a single camera 
capable of movement between two predetermined points for obtaining images at two 
predetermined angles of view. It is important that cameras 14a, 16a, 114, 214, 116, 
216, and 14b, 16b are positioned so that the user's face is within the focusing area of 
the respective cameras when in a position to access the system. The cameras may 
also be provided with horizontal, vertical and zooming capabilities in order to enable 
accurate obtaining of the images of the user's face. The operation of cameras 14a, 
14b, 114, 214, 116, 216, and 16a, 16b and light source 18a, 18b will be described in 
conjunction with the flow diagram of Figure 6. 

[0017] Upon activation of the computer 20, or upon detection of an attempt to 
access computer 20 of privileged information contained thereon, light source 18a, 18b 
projects structured light onto the user's face (step 30) and each camera 14a, 14b and 
16a, 16b obtains an image of the user's face from its respective angle point of view 
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(step 32). Structured light is a grid of light that provides image reference points for use 
in the surface reconstruction of the 3-dimensional model of the user's face. According 
to the invention, the structured light may be generated by any light source capable of 
being imaged (i.e., seen) by a CCD camera. Examples of such light are, LASER light, 
visible white light, and infrared light. When using LASER or infrared, consideration must 
be made as to the effect, if any, the projected light may have on the user's eyes. Low 
levels and durations of LASER light and infrared light are not dangerous to a person's 
eyes, and as such provide other light source alternatives. 

[0018] The use of structured light also prevents the possibility of trying to use 
mirrors and two-dimensional photographs in an attempt to fool the system into thinking 
the authorized user is present. The structured light prevents such attempts by bending 
with the 3-dimensional contours of the user's face. When mirrors and 2-D photos are 
used, the structured light will not bend on the contours of the user's face, thereby 
indicating to the system that the user is not, in fact, sitting in front of the computer. The 
two cameras take photos of the user from two different predetermined angles with a 
slight delay between the first and second image obtained. This delay is dependent on 
the camera systems used, and for example when using a video camera, may be 
decreased to only a few milliseconds. Figures 3a and 3b show the first and second 
images, respectively, obtained with the structured light projected onto the user's face. 
[0019] Once the images are obtained at step 32, the computer 20 or other 
processing device generates a 3-dimensional model of the user's head (step 34). The 
process of generating a 3-dimensional model therefrom is referred to as surface 
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reconstruction. The 3-dimensional model generated at step 34 is performed using any 
suitable known surface reconstruction algorithm. Examples of such surface 
reconstruction techniques can be found in U.S. Patent Nos. 5,202,928, 5,671,343, 
5,694,483 and 5,819,016, each which are incorporated herein by reference. Figures 
4a-4c show the result of the surface reconstruction as a cloud of data points. The 
accuracy obtained in the cloud of data points can be user prescribed to be better or 
worse depending on the usable processing power (e.g. of computer 20) and the level of 
accuracy desired. 

[0020] Once the 3-dimensional model has been generated, a normal picture is 
taken of the user's face (Figure 5) which provides a facial texture bit map representation 
of the user. The 3D model and the facial texture information of the user are then 
compared with a previously created user profile stored in a memory contained within 
computer 20 (step 40). If the 3-dimensional data points match the points in the profile 
accurately enough and the facial texture bit map sufficiently matches the stored profile, 
access is granted (step 42) to the system or specific files. If the comparison does not 
result in sufficiently positive identification of the user, access is denied (step 44). 
[0021] Other features of the present invention may include a profile adjustment 
procedure performed during the match step 40. An example of when such adjustment 
would be necessary would be when the user's facial shape has changed for reasons 
such as, gaining weight or aging. In these instances, the user's profile may be updated 
during the match step to ensure correct recognition of the user for future applications. 
For example, every time the user is granted access, the system will record the newly 
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scanned model and add it to the previous stored profile such that the current stored 
profile is an averaged profile. Thus, the stored profile will evolve with the changes in 
the user's appearance, and therefore never deny access if the change in the user's 
appearance is not too far from the average profile stored. In the event that the user's 
appearance has changed drastically since their last access attempt, for example, if the 
user has gained 100 pounds, they will be required to contact a system administrator for 
purposes re-programming the system to recognize the user and store a new profile for 
the user. 

[0022] While there have shown and described and pointed out fundamental novel 
features of the invention as applied to preferred embodiments thereof, it will be understood 
that various omissions and substitutions and changes in the form and details of the 
devices illustrated, and in their operation, may be made by those skilled in the art without 
departing from the spirit of the invention. For example, it is expressly intended that all 
combinations of those elements and/or method steps which perform substantially the 
same function in substantially the same way to achieve the same results are within the 
scope of the invention. It is the intention, therefore, to be limited only as indicated by the 
scope of the claims appended hereto. 
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